Overview of the Clef 2008 Multilingual Question Answering Track
Identifieur interne : 000043 ( France/Analysis ); précédent : 000042; suivant : 000044Overview of the Clef 2008 Multilingual Question Answering Track
Auteurs : Pamela Forner [Italie] ; Anselmo Pe As [Espagne] ; Eneko Agirre [Espagne] ; I Aki Alegria [Espagne] ; Corina For Scu [Roumanie] ; Nicolas Moreau [France] ; Petya Osenova [Bulgarie] ; Prokopis Prokopidis [Grèce] ; Paulo Rocha [Portugal] ; Bogdan Sacaleanu [Allemagne] ; Richard Sutcliffe [Irlande (pays)] ; Erik Tjong Kim Sang [Pays-Bas]Source :
- Lecture Notes in Computer Science [ 0302-9743 ] ; 2009.
Abstract
Abstract: The QA campaign at CLEF 2008 [1], was mainly the same as that proposed last year. The results and the analyses reported by last year’s participants suggested that the changes introduced in the previous campaign had led to a drop in systems’ performance. So for this year’s competition it has been decided to practically replicate last year’s exercise. Following last year’s experience some QA pairs were grouped in clusters. Every cluster was characterized by a topic (not given to participants). The questions from a cluster contained co-references between one of them and the others. Moreover, as last year, the systems were given the possibility to search for answers in Wikipedia as document corpus beside the usual newswire collection. In addition to the main task, three additional exercises were offered, namely the Answer Validation Exercise (AVE), the Question Answering on Speech Transcriptions (QAST), which continued last year’s successful pilots, together with the new Word Sense Disambiguation for Question Answering (QA-WSD). As general remark, it must be said that the main task still proved to be very challenging for participating systems. As a kind of shallow comparison with last year’s results the best overall accuracy dropped significantly from 42% to 19% in the multi-lingual subtasks, but increased a little in the monolingual sub-tasks, going from 54% to 63%.
Url:
DOI: 10.1007/978-3-642-04447-2_34
Affiliations:
- Allemagne, Bulgarie, Espagne, France, Grèce, Irlande (pays), Italie, Pays-Bas, Portugal, Roumanie
- Groningue (province)
- Groningue (ville), Paris
- Université de Groningue
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 002075
- to stream Istex, to step Curation: 001C44
- to stream Istex, to step Checkpoint: 000725
- to stream Main, to step Merge: 000C64
- to stream Main, to step Curation: 000C54
- to stream Main, to step Exploration: 000C54
- to stream France, to step Extraction: 000043
Links to Exploration step
ISTEX:45E3DE933BAC8DF1750743AE46EADF865D590546Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Overview of the Clef 2008 Multilingual Question Answering Track</title><author><name sortKey="Forner, Pamela" sort="Forner, Pamela" uniqKey="Forner P" first="Pamela" last="Forner">Pamela Forner</name></author><author><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></author><author><name sortKey="Agirre, Eneko" sort="Agirre, Eneko" uniqKey="Agirre E" first="Eneko" last="Agirre">Eneko Agirre</name></author><author><name sortKey="Alegria, I Aki" sort="Alegria, I Aki" uniqKey="Alegria I" first="I Aki" last="Alegria">I Aki Alegria</name></author><author><name sortKey="For Scu, Corina" sort="For Scu, Corina" uniqKey="For Scu C" first="Corina" last="For Scu">Corina For Scu</name></author><author><name sortKey="Moreau, Nicolas" sort="Moreau, Nicolas" uniqKey="Moreau N" first="Nicolas" last="Moreau">Nicolas Moreau</name></author><author><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name></author><author><name sortKey="Prokopidis, Prokopis" sort="Prokopidis, Prokopis" uniqKey="Prokopidis P" first="Prokopis" last="Prokopidis">Prokopis Prokopidis</name></author><author><name sortKey="Rocha, Paulo" sort="Rocha, Paulo" uniqKey="Rocha P" first="Paulo" last="Rocha">Paulo Rocha</name></author><author><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></author><author><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></author><author><name sortKey="Tjong Kim Sang, Erik" sort="Tjong Kim Sang, Erik" uniqKey="Tjong Kim Sang E" first="Erik" last="Tjong Kim Sang">Erik Tjong Kim Sang</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:45E3DE933BAC8DF1750743AE46EADF865D590546</idno><date when="2009" year="2009">2009</date><idno type="doi">10.1007/978-3-642-04447-2_34</idno><idno type="url">https://api.istex.fr/document/45E3DE933BAC8DF1750743AE46EADF865D590546/fulltext/pdf</idno><idno type="wicri:Area/Istex/Corpus">002075</idno><idno type="wicri:Area/Istex/Curation">001C44</idno><idno type="wicri:Area/Istex/Checkpoint">000725</idno><idno type="wicri:doubleKey">0302-9743:2009:Forner P:overview:of:the</idno><idno type="wicri:Area/Main/Merge">000C64</idno><idno type="wicri:Area/Main/Curation">000C54</idno><idno type="wicri:Area/Main/Exploration">000C54</idno><idno type="wicri:Area/France/Extraction">000043</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Overview of the Clef 2008 Multilingual Question Answering Track</title><author><name sortKey="Forner, Pamela" sort="Forner, Pamela" uniqKey="Forner P" first="Pamela" last="Forner">Pamela Forner</name><affiliation wicri:level="1"><country xml:lang="fr">Italie</country><wicri:regionArea>CELCT, Trento</wicri:regionArea><wicri:noRegion>Trento</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Italie</country></affiliation></author><author><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name><affiliation wicri:level="1"><country xml:lang="fr">Espagne</country><wicri:regionArea>Departamento de Lenguajes y Sistemas Informáticos, UNED, Madrid</wicri:regionArea><wicri:noRegion>Madrid</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Espagne</country></affiliation></author><author><name sortKey="Agirre, Eneko" sort="Agirre, Eneko" uniqKey="Agirre E" first="Eneko" last="Agirre">Eneko Agirre</name><affiliation wicri:level="1"><country xml:lang="fr">Espagne</country><wicri:regionArea>Computer Science Department, University of Basque Country</wicri:regionArea><wicri:noRegion>University of Basque Country</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Espagne</country></affiliation></author><author><name sortKey="Alegria, I Aki" sort="Alegria, I Aki" uniqKey="Alegria I" first="I Aki" last="Alegria">I Aki Alegria</name><affiliation wicri:level="1"><country xml:lang="fr">Espagne</country><wicri:regionArea>University of Basque Country</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Espagne</country></affiliation></author><author><name sortKey="For Scu, Corina" sort="For Scu, Corina" uniqKey="For Scu C" first="Corina" last="For Scu">Corina For Scu</name><affiliation wicri:level="1"><country xml:lang="fr">Roumanie</country><wicri:regionArea>UAIC and RACAI</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Roumanie</country></affiliation></author><author><name sortKey="Moreau, Nicolas" sort="Moreau, Nicolas" uniqKey="Moreau N" first="Nicolas" last="Moreau">Nicolas Moreau</name><affiliation wicri:level="1"><country xml:lang="fr">France</country><wicri:regionArea>ELDA/ELRA, Paris</wicri:regionArea><placeName><settlement type="city">Paris</settlement></placeName></affiliation><affiliation><wicri:noCountry code="no comma">E-mail: moreau@elda.org</wicri:noCountry></affiliation></author><author><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name><affiliation wicri:level="1"><country xml:lang="fr">Bulgarie</country><wicri:regionArea>BTB</wicri:regionArea></affiliation><affiliation><wicri:noCountry code="no comma">E-mail: petya@bultreebank.org</wicri:noCountry></affiliation></author><author><name sortKey="Prokopidis, Prokopis" sort="Prokopidis, Prokopis" uniqKey="Prokopidis P" first="Prokopis" last="Prokopidis">Prokopis Prokopidis</name><affiliation><wicri:noCountry code="subField">Athena Research Center</wicri:noCountry></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Grèce</country></affiliation></author><author><name sortKey="Rocha, Paulo" sort="Rocha, Paulo" uniqKey="Rocha P" first="Paulo" last="Rocha">Paulo Rocha</name><affiliation wicri:level="1"><country xml:lang="fr">Portugal</country><wicri:regionArea>Linguateca, DEI UC</wicri:regionArea><wicri:noRegion>DEI UC</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Portugal</country></affiliation></author><author><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name><affiliation wicri:level="1"><country xml:lang="fr">Allemagne</country><wicri:regionArea>DFKI</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Allemagne</country></affiliation></author><author><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name><affiliation wicri:level="1"><country xml:lang="fr">Irlande (pays)</country><wicri:regionArea>DLTG, University of Limerick</wicri:regionArea><wicri:noRegion>University of Limerick</wicri:noRegion></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Irlande (pays)</country></affiliation></author><author><name sortKey="Tjong Kim Sang, Erik" sort="Tjong Kim Sang, Erik" uniqKey="Tjong Kim Sang E" first="Erik" last="Tjong Kim Sang">Erik Tjong Kim Sang</name><affiliation wicri:level="4"><country>Pays-Bas</country><placeName><settlement type="city">Groningue (ville)</settlement><region>Groningue (province)</region></placeName><orgName type="university">Université de Groningue</orgName></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Pays-Bas</country></affiliation></author></analytic><monogr></monogr><series><title level="s">Lecture Notes in Computer Science</title><imprint><date>2009</date></imprint><idno type="ISSN">0302-9743</idno><idno type="eISSN">1611-3349</idno></series><idno type="istex">45E3DE933BAC8DF1750743AE46EADF865D590546</idno><idno type="DOI">10.1007/978-3-642-04447-2_34</idno><idno type="ChapterID">Chap34</idno><idno type="ChapterID">34</idno></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass><langUsage><language ident="en">en</language></langUsage></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: The QA campaign at CLEF 2008 [1], was mainly the same as that proposed last year. The results and the analyses reported by last year’s participants suggested that the changes introduced in the previous campaign had led to a drop in systems’ performance. So for this year’s competition it has been decided to practically replicate last year’s exercise. Following last year’s experience some QA pairs were grouped in clusters. Every cluster was characterized by a topic (not given to participants). The questions from a cluster contained co-references between one of them and the others. Moreover, as last year, the systems were given the possibility to search for answers in Wikipedia as document corpus beside the usual newswire collection. In addition to the main task, three additional exercises were offered, namely the Answer Validation Exercise (AVE), the Question Answering on Speech Transcriptions (QAST), which continued last year’s successful pilots, together with the new Word Sense Disambiguation for Question Answering (QA-WSD). As general remark, it must be said that the main task still proved to be very challenging for participating systems. As a kind of shallow comparison with last year’s results the best overall accuracy dropped significantly from 42% to 19% in the multi-lingual subtasks, but increased a little in the monolingual sub-tasks, going from 54% to 63%.</div></front></TEI><affiliations><list><country><li>Allemagne</li><li>Bulgarie</li><li>Espagne</li><li>France</li><li>Grèce</li><li>Irlande (pays)</li><li>Italie</li><li>Pays-Bas</li><li>Portugal</li><li>Roumanie</li></country><region><li>Groningue (province)</li></region><settlement><li>Groningue (ville)</li><li>Paris</li></settlement><orgName><li>Université de Groningue</li></orgName></list><tree><country name="Italie"><noRegion><name sortKey="Forner, Pamela" sort="Forner, Pamela" uniqKey="Forner P" first="Pamela" last="Forner">Pamela Forner</name></noRegion><name sortKey="Forner, Pamela" sort="Forner, Pamela" uniqKey="Forner P" first="Pamela" last="Forner">Pamela Forner</name></country><country name="Espagne"><noRegion><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></noRegion><name sortKey="Agirre, Eneko" sort="Agirre, Eneko" uniqKey="Agirre E" first="Eneko" last="Agirre">Eneko Agirre</name><name sortKey="Agirre, Eneko" sort="Agirre, Eneko" uniqKey="Agirre E" first="Eneko" last="Agirre">Eneko Agirre</name><name sortKey="Alegria, I Aki" sort="Alegria, I Aki" uniqKey="Alegria I" first="I Aki" last="Alegria">I Aki Alegria</name><name sortKey="Alegria, I Aki" sort="Alegria, I Aki" uniqKey="Alegria I" first="I Aki" last="Alegria">I Aki Alegria</name><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></country><country name="Roumanie"><noRegion><name sortKey="For Scu, Corina" sort="For Scu, Corina" uniqKey="For Scu C" first="Corina" last="For Scu">Corina For Scu</name></noRegion><name sortKey="For Scu, Corina" sort="For Scu, Corina" uniqKey="For Scu C" first="Corina" last="For Scu">Corina For Scu</name></country><country name="France"><noRegion><name sortKey="Moreau, Nicolas" sort="Moreau, Nicolas" uniqKey="Moreau N" first="Nicolas" last="Moreau">Nicolas Moreau</name></noRegion></country><country name="Bulgarie"><noRegion><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name></noRegion></country><country name="Grèce"><noRegion><name sortKey="Prokopidis, Prokopis" sort="Prokopidis, Prokopis" uniqKey="Prokopidis P" first="Prokopis" last="Prokopidis">Prokopis Prokopidis</name></noRegion></country><country name="Portugal"><noRegion><name sortKey="Rocha, Paulo" sort="Rocha, Paulo" uniqKey="Rocha P" first="Paulo" last="Rocha">Paulo Rocha</name></noRegion><name sortKey="Rocha, Paulo" sort="Rocha, Paulo" uniqKey="Rocha P" first="Paulo" last="Rocha">Paulo Rocha</name></country><country name="Allemagne"><noRegion><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></noRegion><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></country><country name="Irlande (pays)"><noRegion><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></noRegion><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></country><country name="Pays-Bas"><region name="Groningue (province)"><name sortKey="Tjong Kim Sang, Erik" sort="Tjong Kim Sang, Erik" uniqKey="Tjong Kim Sang E" first="Erik" last="Tjong Kim Sang">Erik Tjong Kim Sang</name></region><name sortKey="Tjong Kim Sang, Erik" sort="Tjong Kim Sang, Erik" uniqKey="Tjong Kim Sang E" first="Erik" last="Tjong Kim Sang">Erik Tjong Kim Sang</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Musique/explor/MozartV1/Data/France/Analysis
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000043 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/France/Analysis/biblio.hfd -nk 000043 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Musique
|area= MozartV1
|flux= France
|étape= Analysis
|type= RBID
|clé= ISTEX:45E3DE933BAC8DF1750743AE46EADF865D590546
|texte= Overview of the Clef 2008 Multilingual Question Answering Track
}}
| This area was generated with Dilib version V0.6.20. |  |